test(evidence): deflake prose/visual asserts against committed registry - #192
Merged
Conversation
test_readme_distinguishes_every_registered_token_context_measurement, test_context_savings_visual_uses_only_registered_measurements and test_benchmark_guide_tracks_the_live_offline_evaluators interpolated values from a LIVE evaluator run and compared them to prose written from the committed registry — a race between wall-clock sampling variance (observed 23,808 vs 23,810 payload tokens; 85.35 vs 85.38 mean) and the published decimals. Prose/visual assertions now interpolate from the COMMITTED registry artifact via _committed_evidence(), making them deterministic. The live-vs-registry bridge (test_public_numeric_evidence_registry...) keeps exact equality for integers/categorical fields and applies a 0.5% relative tolerance only to timing-derived aggregates (mean/max tokens, payload totals, ratio), so real drift still fails while scheduler jitter does not.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test(evidence): deflake prose/visual asserts against committed registry
Follow-up to the flaky
test_benchmark_evidencefailures observed on main after #190: the registry-vs-live comparison and the prose/visual assertions interpolated values from a LIVE evaluator run whose timing-derived aggregates wobble run-to-run (observed 23,808 vs 23,810 payload tokens; 85.35 vs 85.38 mean context).Changes (1 commit on top of #191-merged main)
docs/benchmark-evidence/offline-fixtures-v1.json) via a_committed_evidence()helper — README, BENCHMARKS and both SVGs were written from that artifact, so assertions can't drift from the evidence they cite:test_readme_distinguishes_every_registered_token_context_measurementtest_context_savings_visual_uses_only_registered_measurementstest_benchmark_guide_tracks_the_live_offline_evaluatorstest_public_numeric_evidence_registry_is_complete_and_live) keeps exact equality for integer/categorical fields and applies a 0.5% relative tolerance only to timing-derived aggregates (mean/max tokens, payload totals, ratio) — real drift still fails, scheduler jitter does not.Verification
test_benchmark_evidence: 36 passed, repeatedly (previously flaked 4/36)ruffclean8a74e9f4…still correct)